Skip to content

Add BigQuery CREATE EXTERNAL TABLE WITH PARTITION COLUMNS parsing support#66

Open
mesmacosta wants to merge 2 commits intotobilg:mainfrom
mesmacosta:feat/bigquery-create-external-table
Open

Add BigQuery CREATE EXTERNAL TABLE WITH PARTITION COLUMNS parsing support#66
mesmacosta wants to merge 2 commits intotobilg:mainfrom
mesmacosta:feat/bigquery-create-external-table

Conversation

@mesmacosta
Copy link
Copy Markdown

Summary

BigQuery's CREATE EXTERNAL TABLE syntax supports clauses like WITH PARTITION COLUMNS and WITH CONNECTION that use the WITH keyword differently from the generic WITH properties parser. The generic parser was causing parse failures whenever column definitions or partition columns were present.

Example: failing query before this fix

CREATE EXTERNAL TABLE IF NOT EXISTS `{project}.{dataset}.{table}`
WITH PARTITION COLUMNS (
  table_name  STRING,
  sync_date   DATE,
  start_date  DATE,
  end_date    DATE,
  sync_id     STRING
)
OPTIONS (
  format                        = 'PARQUET',
  uris                          = ['gs://{bucket}/data/table_name={table}/*'],
  hive_partition_uri_prefix     = 'gs://{bucket}/data',
  require_hive_partition_filter = false
);
Error: Expected identifier, got LParen

Fix:

  • Consolidate all BigQuery EXTERNAL TABLE WITH handling into a single location at the WITH token consumption site, with a clean early return
  • Add EXTERNAL to BigQuery's special_modifier dialect list
  • Add generation support (builder + generator) for the new AST fields (with_partition_columns, with_connection)

This does not affect parsing of regular BigQuery CREATE TABLE statements, CTEs in AS SELECT, or any other dialect's WITH handling — the logic only activates when table_modifier == "EXTERNAL" AND dialect == BigQuery.

Changes

  • parser.rs — BigQuery EXTERNAL TABLE parsing + 19 new tests
  • expressions.rs — New with_partition_columns and with_connection fields on CreateTable
  • builder.rs — Builder support for new fields
  • generator.rs — SQL generation for BigQuery EXTERNAL TABLE syntax
  • dialects/mod.rs — Add EXTERNAL to BigQuery's special modifiers

Test plan

  • 12 new BigQuery EXTERNAL TABLE tests (basic, with columns, with partition columns, with connection, full, roundtrips, IF NOT EXISTS, OR REPLACE, PR description SQL)
  • 8 new BigQuery WITH syntax compatibility tests (regular AS SELECT, single CTE, multiple CTEs, PARTITION BY + CLUSTER BY + OPTIONS, OR REPLACE, IF NOT EXISTS)
  • All 20 BigQuery-specific tests pass
  • Full test suite: 917 passed, 0 failed, 0 regressions

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants